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ABSTRACT 
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referred to as the "Science: Parents, Activities, and Literature" (Science 
PALs) program focused on incorporating children's literature and parent 
partners using an interactive-constructivist epistemology as its base. 
Individual teachers were rated by the school district's science coordinator 
on their use of specific Science PALs strategies. Student attitudes and 
awareness were assessed with a paper and pencil survey developed for this 
research. Student achievement was assessed with multiple choice and 
constructed response items taken from the released item pool of the Third 
International Mathematics and Science Study (TIMSS) . Student attitude, 
awareness, and achievement data were analyzed in terms of the cumulative 
ratings of teachers across the last three years of the reform effort. Results 
indicate that students receiving three years of instruction from teachers 
with different ratings on their use of Science PALs strategies did not show 
significantly different attitude, awareness, or achievement. These results 
are discussed in terms of school district characteristics, research design 
factors, and challenges of systemic reform goals. (Contains 19 references.) 
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Abstract 



This study explored the cumulative effect of three years of exposure to teaching strategies 
being promoted in a local school district systemic reform on elementary students' attitudes 
toward school science, awareness of science careers, and science achievement. The reform effort, 
referred to as the Science: Parents, Activities and Literature (Science PALs) program focused on 
incorporating children’s literature and parent partners using an interactive-constructivist 
epistemology as its base. Individual teachers were rated by the school district’s science 
coordinator on their use of specific Science PALs strategies. Student attitudes and awareness 
were assessed with a paper and pencil survey developed for this research. Student achievement 
was assessed with multiple choice and constructed response items taken from the released item 
pool of the Third International Mathematics and Science Study (TIMSS). Student attitude, 
awareness, and achievement data were analyzed in terms of the cumulative ratings of teachers 
across the last three years of the reform effort. Results indicate that students receiving three years 
of instruction from teachers with different ratings on their use of Science PALs strategies did not 
show significantly different attitude, awareness, or achievement. These results are discussed in 
terms of school district characteristics, research design factors, and challenges of systemic 
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A Study of Changes in Students’ Science Attitudes, Awareness and Achievement Across 
Three Years as a Function of the Level of Implementation of Interactive-Constructivist 
Teaching Strategies Promoted in a Local Systemic Reform Effort 



James A. Shymansky Larry D. Yore John O. Anderson 

University of Missouri-St. Louis University of Victoria University of Victoria 

Introduction 

This paper is about one school district’s effort to effect a major reform of its elementary 
science curriculum. The district had an extensive hands-on, kit-based elementary school science 
curriculum in place. The curriculum was supported by a district science coordinator and a 
materials distribution center. The kits contained exemplary National Science Foundation (NSF) 
supported materials from recent curriculum projects and others based on the 1960s projects. The 
kits were delivered to the district teachers on a rotating basis. While the students enjoyed doing 
activities, there was a strong sense among the teachers and district staff that students were not 
developing meaningful science understandings from the experiences. It was generally 
acknowledged that the typical elementary school teacher in the district had little understanding of 
the science concepts explored in the kits, was uncomfortable teaching science, and believed that 
simply doing activities would enhance students’ science process and thinking skills. There was 
little or no attention paid to developing specific conceptual knowledge or to addressing students’ 
mis-conceptual knowledge. The Science: Parents, Activities and Literature (Science PALs) 
project was launched in 1994 to address the district's concerns with its elementary science 
program. 

Problem 

Science PALs stressed the importance of children’s ideas about science, strategies for 
challenging children’s prior knowledge to stimulate conceptual growth and change, connections 
to other content disciplines in the children’s school day, and parents as partners in their 
children’s science education. Since all these elements ultimately play out in the classroom, it was 
decided that the program’s impact on the students should be considered as the primary measure 
of the project's success. But since Science PALs was a multi-year project, it was further decided 
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to explore the cumulative impact of the reformed science program on the relatively untainted 
student population of the early elementary school grades. Thus, this study focused on students' 
science learning over a three-year period using Grade 3 and Grade 4 students who were in 
Kindergarten and Grade 1 at the outset of this study. Students' attitudes toward science, 
awareness of science careers, and science achievement were explored as a function of the quality 
of the implementation of a Science PALs brand of instruction over three years. Quality of 
implementation was measured by the science coordinator's ratings of teachers using a four- 
dimension rubric. Science attitudes and science career awareness were measured with Likert- 
type items developed specifically for this study and science achievement was measured by a 
TIMSS-based instrument using released items for Grades 3 and 4. 

Background 

Science teaching, science learning, and science teacher education research has attracted 
increased attention and funding in recent years with the publication of the National Research 
Council’s National Science Education Standards (NRC, 1996), the National Board for 
Professional Teaching Standards (NBPTS, 1994), and the Report of the National Commission on 
Teaching and America’s Future (Darling-Hammond, 1996). These reform documents reaffirm 
the importance of teachers, teaching, and hands-on/minds-on learning as primary influences on 
students’ thinking, achievement, and science literacy. Collectively, the documents provide a 
vision of what we should teach, how we should teach, and how we should teach teachers to 
teach. 

Furthermore, an analysis of the reform documents for language arts, mathematics, 
science, social studies, and technology revealed a common focus on “all” students, common 
learning outcomes of literacy and critical thinking, and common instructional intentions 
regarding constructivism and authentic assessment (Ford, Yore, & Anthony, 1997). Science 
literacy, like the other contemporary literacies, involves critical thinking, cognitive abilities and 
habits-of-mind to construct understanding in the specific disciplines, the big ideas or the unifying 
concepts of the disciplines and the communication skills to share these understandings and to 
persuade others to take informed action. 

Unfortunately, little attention has been given to developing a concise, clear definition of 
constructivism and of the associated classroom practices (NRC, 1996, p. 52). The National 
Science Teachers Association (1997) encourages teachers to increase their awareness of the 

O 

ERIC 



4 



emerging standards for teaching, professional development, assessment, content, program, and 
the full education system. Clearly, proponents of the current science reform believes it is not 
enough to specify learning outcomes without emphasizing the quality of the learning experience, 
the authenticity of the evaluation, and the availability of learning opportunities. 

The interactive-constructivist brand of science teaching promoted in the Science PALs 
project is a middle-of-the-road interpretation of constructivism (Shymansky, et al, 1997). 
Interactive-constructivist teaching recognizes a specific worldview of thinking, the 
epistemological and ontological nature of science, the locus of mental activity in the learner, the 
socio-cultural aspects of the classroom, the multiple purposes of language, and the realities of 
public education and schools (Yore & Shymansky, 1997). Interactive-constructivist science 
teaching assumes that contemporary science is based on a hybrid view of knowing that stresses 
the importance of interactions with the physical world and the sociocultural context in which 
interpretations of these experiences reflect the lived experiences and cultural beliefs of the 
knowers (Prawat & Floden, 1994). 

An interactive-constructivist perspective also assumes an epistemological and ontological 
view of science that stresses evaluation of knowledge. This evaluation requires that explanations 
and interpretations are judged against the available data and canonical theories using evidence 
from Nature and scientific warrants to justify claims about reality (Hofer & Pintrich, 1997; 
Kuhn, 1993). The locus of mental activity and construction of understanding in interactive- 
constructivism involves both a private and public component, unlike social constructivism which 
defines understanding as a group consensus building or radical constructivism which defines 
understanding as a uniquely individual decision (Hennessey, 1994; Prawat & Floden, 1994). An 
interactive-constructivist perspective assumes that discourse reveals the variety of alternative 
interpretations but that consensus need not be reached. It is evidence from Nature that supports 
or rejects the interpretations, not consensus (Fosnot, 1996; Prawat & Floden, 1994). The locus of 
control for learning in an interactive-constructive model is shared by the learner and the teacher. 
The basic assumptions about the role of prior knowledge, the plausibility of alternative ideas, and 
the resiliency of these ideas are preserved in an interactive-constructivist perspective; but 
professional wisdom, the accountability of public education, and the priorities of elementary 
schools mediate decisions about what to teach and how to teach in the science classroom. 



The Science PALs Project 



The Science PALs project is an on-going local systemic reform effort that was originally 
funded, by the National Science Foundation and the Howard Hughes Medical Institute between 
1994 and 1998. During the funded period K-6 teachers and Grades 7-12 science teachers 
participated in summer workshops and school year inservice activities. In addition to the 
teachers, approximately 3400 parents participated in special training sessions designed to 
integrate them into the K-6 science instruction. Across the four years of Science PALs, teachers 
received an average of 110 hours of inservice education to enhance their pedagogical-content 
knowledge. 

The first year of the Science PALs Project began with 16 elementary school teachers 
designated as science advocates — one from each elementary school in the district. The science 
advocates began the project by attending a special, problem-centered summer workshop. That 
workshop was designed to help participants explore selected NSF-sponsored curriculum units 
and activities using students’ ideas as “straw men”. The workshop matched science content 
consultants with small groups of science advocates to explore science concepts in the units 
selected by grade level groups. Interactive-constructivist teaching strategies focusing on the use 
of students' ideas, literature, and parent partners were modeled and taught in the workshop. 

The summer workshop with the follow-up inservice cycle was expanded in the next three 
project years involving approximately 40 teachers in the second year, 80 teachers in the third 
year, and 140 teachers in the fourth year. The inservice cycle focused on authentic problems of 
curriculum adaptation, using activities to challenge teachers’ ideas, and social interactions and 
private reflections to get the teachers to rethink their ideas about both the science content of the 
units and how to teach the units. A similar instructional cycle was then used by the teachers to 
challenge their students’ ideas and to promote conceptual growth and change in science. 

Design 

The research questions for this study were addressed using a comparative groups design 
that utilized attitude, awareness, and achievement data collected on all 1998 Grade 3 and Grade 4 
students as dependent variables and cumulative ratings of their 1998, 1997, and 1996 teachers on 
their use of specific Science PALs strategies as the independent variable. The independent 
variable in the study effectively represented a measure of the “three-year implementation 
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quality” of district teachers’ use of student ideas, children’s literature, and parent partners in 
teaching science. Scores from a specially constructed student attitudes and awareness survey and 
a modified TIMSS provided data on the dependent variables — attitudes toward school science, 
awareness of science careers, and science achievement. 

Independent Variable 

Implementation quality of the Science PALs brand of interactive-constructivist 
instruction was rated by the school district science coordinator. All K-6 teachers with a science 
teaching responsibility were rated on an 4-dimension rubric developed to assess the unique 
features of the Science PALs interactive-constructivist approach. The rubric required the 
coordinator to assess (1-very weak, 2-weak, 3-satisfactory, 4-strong, 5-very strong) the degree of 
compliance on dimensions stressed in the reform effort: 

1. Use of strategies to access and utilize information on student ideas in planning 
instruction. 

2. Use of strategies to challenge student ideas and to have them reflect on and integrate 
those ideas into their thinking. 

3. Use of strategies that routinely and continuously incorporate children's literature and 
personal experiences as context for learning science. 

4. Use of strategies that promote ongoing, substantive parent involvement in the science 
instruction. 

The substantive validity, external validity, structural validity, and reliability of the rubric 
were established by a series of inquiries (Messick, 1989). First, the substantive validity was 
established by examining the degree to which dimensions in the rating scale matched both the 
theoretical and practical assumptions and goals of the project. This match was verified further 
with correlation studies on the rubric dimensions 1, 2, 3, and 4. Correlations between the 
individual pairs of items were between 0.68 and 0.95 for the 1997 ratings and between 0.78 and 
0.95 for the 1998 ratings. 

The external validity was explored by t-test analyses of 128 teachers' ratings over a 
complete Science PALs cycle. Comparisons of the 1996 and the 1997 clustered Science PALs 
dimension revealed statistically significant predicted improvement (t=5.0, p-value <0.001). 

The structural validity was checked by a series of factor analyses. The one-factor solution 
was supportive (component loadings of 0.80 to 0.97) indicating that the 4 dimensions were 
acting as a unified factor. 
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Reliability was established by the correlation results and by a rate/re -rate analysis of a 
random sample of teachers. Measures of internal consistency for the 4-dimension cluster were 
0.96 (1997 ratings) and 0.97 (1998 ratings). The science coordinator's rating consistency was 
explored by asking her to rate 235 teachers who teach science in the 16 elementary schools. One 
week later a random sample of 20 teachers were re-rated by the science coordinator on the ploy 
that their rating results were lost. The correlations of these paired ratings was 0.95. 

Dependent Variables 

An five-position Likert scale survey designed to assess students' strong agreement (4.0), 
agreement (3.0), absence of opinion (2.5), agreement (2.0), or strong disagreement (1.0) on 
statements describing students’ affective stance toward school science and science careers was 
used to measure students' attitudes and awareness (Dunkhase, et al, 1997; Shymansky, et al, 
1998; Yore, et al, 1998). Validity and reliability of this survey were explored using expert 
analysis, factor analyses, and internal consistency. The construct validity of the instrument was 
investigated by having experts examine the items selected from established item pools or 
constructed by science educators for the project. Factor analyses were conducted on the 1996 
results from 722 Grade 3 and Grade 4 students taking the original pool of items. The final 
version of the instrument was then constructed using only items that had factor loadings of 
greater than 0.30. The items retained were analyzed again to insure the resulting factors matched 
the design features of the instrument. The 2 factors in the attitudes toward science scale were 
identified as attitudes toward school science and awareness of science careers. Internal 
consistency of the combined scale was 0.79, of the 5-item school science sub-scale, 0.74, and of 
the 3-item science careers sub-scale, 0.72. The same instrument administered to 456 students the 
next year revealed internal consistencies of 0.75, 0.74, and 0.69 for the overall survey and the 
two sub-scales. 

Released items from the TIMSS Grades 3-4 were collected into 6 tests to assess science 
achievement (IEA, 1997). Each form of the science achievement test contained 29 items: 22 
multiple-choice and 7 constructed-response (short-answer and extended-response). All forms of 
the test had 9 common multiple choice items and 3 common short-answer items. Various 
combinations of the forms had other common short-answer items. 

The validity of the special TIMSS forms was assumed to be reasonable since the items 
were selected from those used in the international study. The internal consistency of the 6 forms 
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ranged from 0.73-0.77 (average of 0.75). The 6 forms were randomly distributed across students 
in each classroom tested. Statistical analysis of the results on the common items across the 6 
subgroups revealed no significant differences. Based on this evidence, it was assumed that the 
results across all forms of the test could be combined using standard scores (Z = 50 + lOz-score, 
SD = lOsd) to adjust results for the uncommon composition. The scores for the multiple-choice 
and open-end (short-answer and extended-response) sub-scales might be considered as measures 
of lower-level and higher-level knowledge. 

Students were asked to identify their current year science teacher and their science 
teachers from the previous two years. Knowing a student’s teachers for three consecutive years 
provided the opportunity to examine links between the cumulative quality of science instruction 
and student performance over three years. The district coordinator ratings of the teachers allowed 
linkage of the current year teacher to his/her students directly. The coordinator ratings of all three 
teachers were combined to yield a cumulative quality of Science PALs instruction (QOSPI) 
rating score for each student. The QOSPI allowed examination the cumulative effect of different 
qualities of science teaching across a three-year period on student performance. The three-year 
QOSPI scores could range from 12 to 60. QOSPI scores of 46-60 were categorized as “high” 
(HQOSPI), total ratings of 30-45 were categorized as “medium” (MQOSPI), and total ratings of 
12-29 were categorized as “low” (LQOSPI). 

Results 

Table 1 contains summary descriptive statistics of student attitudes, awareness, and 
achievement grouped by quality of Science PALs instruction (QOSPI). Table 2 provides the 
summary ANOVA results of student attitudes, awareness, and achievement in science as a 
function of QOSPI across three years. 

Table 1 : Descriptive Statistics for Students' Attitudes, Awareness and Achievement Grouped by 
HQOSPI (High), MQOSPI (Middle), and LQOSPI (Low) Ratings 

Teaching Attitude toward Science Careers TIMSS TIMSS 

Experience School Science Awareness Multiple Choice Constructed-Response 

2.89(0.64) 2.55(0.87) 50.8(10.3) 49.2(10.18) 

N = 62 N = 62 N = 90 . N = 52 

2.98(0.65) 2.65(0.73) 51.0(9.69) 50.1(9.94) 
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N = 477 



N = 477 



N = 487 



N = 453 



Low 3.09(0.64) 2.65(0.87) 50.4(9.46) 51.6(9.17) 

N = 35 N = 35 N = 105 N = 50 



Table 2: One-way ANOVA Results for HQOSPI, MQOSPI, and LQOSPI Student Attitudes, 



Awareness and Achievement Scores 



Main Effect 


F ratio 


Degrees of Freedom 


P value 


Attitude toward School Science 


1.15 


2,571 


0.316 


Science Careers Awareness 


0.46 


2,571 


0.630 


TIMSS Multiple Choice 


0.19 


2,681 


0.826 


TIMSS Open-ended 


0.76 


2,552 


0.466 



The analyses yielded no significant differences in student attitudes, awareness or 
achievement among students experiencing different qualities of instruction as defined by the 
teachers’ use of student ideas, children’s literature, and parent partners. 

Discussion 

Results from earlier studies suggest that the Science PALs reform effort was successful in 
several ways. Teachers seemed to embrace the key element of interactive-constructivist 
teaching — they paid more attention to student ideas in science — the district science coordinator 
noted this and so did the students. Teachers also began utilizing children’s literature in their 
science classrooms — again, both the science coordinator and the students noticed this. Finally, 
parents responded extremely positively to their new roles as partners and students sensed this 
increased involvement (Shymansky, Yore & Anderson, 1999; Shymansky, Yore & Hand, 2000). 
These positive results of the earlier studies make the non-significant results of the current study 
seem almost anomalous. All the conditions seemed to be in place to yield positive student 
performance results — more positive student attitudes toward science, greater student awareness 
of and positive dispositions toward careers in science, and higher student science achievement 
scores. So why the non-significant student outcomes? 
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First, an examination of the student achievement results. One explanation for the non- 
significant results is that the elements critical to Science PALs (focus on student ideas, use of 
children’s literature, and parent partners), though positively viewed by students, parents, the 
science coordinator and the teachers themselves, have little to do with how well students leam 
science. While there was good “cardiac” evidence that the Science PALs reform effort was 
effective (i.e., in their hearts, everyone in the district knew the program was good), student 
achievement data simply did not support that good feeling. 

There are, however, at least two other explanations for the non-significant student 
achievement results. The variation of the TIMSS instrument used to assess achievement may not 
have been tuned sufficiently to the school district’s science curriculum to serve as a valid 
measure of achievement. The TIMSS instrument was chosen for its visibility and the existence of 
national and international norms against which the school district could gauge its own students. 
It was also thought that the instrument did not favor students whose teachers were differentially 
effective at embracing and implementing the elements critical to Science PALs. It is interesting 
to note, however, that teachers in the school district did not have any input into the selection of 
items for the instrument constructed from the released TIMSS item pool. We are planning to 
involve a number of the district teachers in a retrospective “item-fitting” activity in which 
teachers will be asked to match TIMSS items on the test with the district’s science units. To 
ensure that this item-fitting is done without bias, student scores on the TIMSS items will not be 
revealed. 

Another explanation for the non-significant achievement results is that student 
achievement in this particular small, university-dominated district are positively influenced by 
forces 'and factors outside the formal school program. Home and community support for 
education and involvement in the schools are very high. As a result, students in this district 
traditionally score very well on standardized test. For example, students in this district routinely 
rank in the top three on the Iowa Tests of Basic Skills and Iowa Test of Educational 
Development. The district also traditionally produces about 20-25 national merit scholars each 
year — more than some states. This combination of supportive parents and an above-average 
student population could account for the smaller variance in the TIMSS scores, making 
differences more difficult to detect among the target groups in this study. 
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Yet a third explanation for the non-significant achievement results could be the quality of 
the district teaching staff and/or the strength of the science program that was in place before the 
Science PALs reform was launched. As mentioned in the introduction, the district did have in 
place a full contingent of hands-on units prior to any inservice instruction on interactive- 
constructivist or Science PALs strategies. It may well be that the district’s elementary science 
program generally and the teachers’ pedagogical content knowledge specifically were 
sufficiently developed to make any improvements in either of these areas small relative to the 
quality of the existing program (a homogenizing effect more than a ceiling effect since there still 
room for improvement in student scores on both the SPOCC and the TIMSS). Since the basic 
science units and supporting kit materials did not change substantially across the four years of 
the Science PALs reform effort and may have been already functioning better than teachers and 
leaders in the district realized, perhaps it should be no surprise that there was no difference in 
student achievement across the groups of teachers. 

The lack of significant results in the student attitudes toward science scores is a little 
more puzzling given the fact that parents were enthusiastic about the Science PALs 
developments and that students saw teachers using their ideas more in class and saw their parents 
more involved in their science instruction. One would think that students receiving three years of 
science instruction from teachers rated the highest on the use of student ideas, children’s 
literature, and parent partners would exhibit the most positive attitudes towards their school 
science. But the results of this study do not show this. Could it again be that we are dealing with 
such small differences between strategies promoted in the Science PALs effort and an already 
effective and positive district hands-on science program that the non-significant differences in 
student attitudes toward science might be expected? After all, the overall mean for student 
attitude toward school science across teacher ratings was just about 3.0 with 4.0 being 
unanimous strong agreement. 

There is one more alternative explanation for the non-significant results in attitude, 
achievement, and career awareness that warrants recognition. It is the time and timing factor. 
This study focuses on the cumulative impact of instruction across the earliest three years of 
school in which science is seen as part of the formal curriculum (at least in the district studied). 
Perhaps the cumulative effect of any set of teaching strategies deemed critical to achievement, 
attitudes, career awareness, etc. does not begin to manifest itself in three years; maybe it simply 



takes more time— four years, five years, or even more? Or perhaps it is a timing thing; younger 
school children may see any hands-on activity in a uniformly positive light, making the impact of 
strategies such as using student ideas and parent partners less remarkable with younger students. 
Perhaps these strategies would be far more powerful with middle school or even high school 
students? 

Concluding Remarks 

The idea of “systemic reform” has been promoted by numerous groups for a little more 
than ten years now. The summer and academic year “institute” models of the 1960s and 1970s 
were seen as too costly and ineffective. It is fairly safe to say that we will not see externally 
funded, one-shot inservice programs targeting single teachers for a long time, if ever again. But 
long-term “systemic reform projects” are no panacea either. State and urban systemic projects of 
the 1990s have produced very little evidence of enhanced teacher knowledge and skills or 
improved student performance. “Local systemic reforms” such as Science PALs likely hold the 
most promise for major, efficient reform of science education, if for no other reason that the 
process of the efforts can be monitored and adjusted relatively easilty and the fruits of the effort 
can be weighed with reasonable accuracy. With a curriculum framework in which to set goals, a 
clear procedure for assessing outcomes and a baseline against which to measure success, local 
systemic reform projects have the potential to produce major, cost-effective improvements in 
how science is taught in our schools. 
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